In the related art, as disclosed in U.S. Pat. No. 8,670,891, an autonomous driving device is known, in which, when autonomous driving is executed in a host vehicle and an amount of any of a steering operation, an acceleration operation or a braking operation by a driver of the host vehicle is equal to or greater than a predetermined threshold value during the autonomous driving, the driving state is switched from autonomous driving to manual driving.